A Question/Answer System, such as the IBM Watson™ system is an artificially intelligent computer system capable of answering questions posed in natural language. One aspect in a QA system is passage support and identifying passages that are relevant to an answer. This aspect is important both from an algorithm standpoint and also to instill confidence in the QA system's answers displayed to the user. One way a QA system identifies relevant passages is based on NLP algorithms. Input to the NLP algorithm is an answer key such that given the Question and Answer—Is the passage relevant? One current way of selecting relevant passages is by using screened Subject Matter Experts (SMEs). The SMEs are people that evaluate a passage and identify whether it provides supporting evidence for the answer. Their feedback is used to produce the answer key and, hopefully, better passage results. This takes time, effort, and investment in the SMEs. Additionally, the involvement of the SMEs occurs primarily during the training and test phase. Once in production, the opportunity to receive value judgments on passages traditionally becomes less controlled. Once the passage is identified as being relevant, it is scored by the QA system. Traditional QA system scoring of a relative passage is somewhat primitive. Within the NLP Scorers, one looks at the Question (Q), the Answer (A), and the passage (P) which was pulled to support the given answer. The passage match is based on similar words, e.g., did the system find that a passage with “George Washington was an American” matched against the Question “Who was the first American President?” In this example, the passage that it was scoring obviously would not completely justify an answer of George Washington for the above Question, but the NLP scorers might have liked it due to the location of ‘American’. Current solutions for scoring the passage relevance asks the user to “rate” the passage as it relates to the question and answer, but ignore the question of whether the SME's judgment should be accepted or if the SME is providing bad data that can poison the QA system's identification of relevant passages given the question and the resulting answer. A further challenge is that the passage ratings are currently not reflected back into the learning algorithm which prevents the QA system from becoming more adept at scoring passages.